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ABSTRACT 


Background: Formation is an important stage of partnership development. Purpose: To describe the systematic ap¬ 
proach to process evaluation of a Turning Point initiative in central Oklahoma during the formation stage. The nine- 
month collaborative effort aimed to develop an action plan to promote health. Methods: A sound planning framework 
was used in the design of a systematic approach to process evaluation. Mixed (qualitative and quantitative) methodol¬ 
ogy was used, including stakeholder interviews, surveys, and attendance logs. Results: Reach to the meetings ranged 
from 38% to 70%. “Collaboration membership” was statistically significantly associated with high attendance at the 
meetings. Strengths of the collaborative process included stakeholder diversity, a strong organizational structure and 
the use of a democratic collaborative process. Discussion: Building effective collaborative skills among the stakeholders 
early in the planning phase can be instrumental in promoting participation during the formation stage. In addition, 
emphasis should be given in strengthening/supporting of the coalition processes, coalition structures, leadership and 
staff. Translation to Health Education Practice: Process evaluation is a valuable tool for the continuous monitoring 
of the quality of the collaborative process during the formation stage, and therefore, minimum process evaluation 
measures should be incorporated at the early stages of the partnership development. 
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accepted for publication on November 19,2010. 


BACKGROUND 

Process evaluation is defined as the mea¬ 
surements obtained during the implementa¬ 
tion of a project in order to “control, assure 
or improve the quality of performance of 
delivery.” 1 Through the collection of both 
qualitative and quantitative data, evalua¬ 
tors can provide continuous feedback to 
the program implementers and assist them 
in modifying components of the interven¬ 
tion to continually enhance its quality. 2 
In the area of partnership and coalition 
development, most process evaluation is 
conducted in the form of a monitoring sys¬ 
tem that tracks both process measures (e.g., 
member participation, planning products, 
media coverage, meetings, budget alloca¬ 


tions) and intermediate measures, such as 
community actions, delivery and ratings of 
satisfaction with the collaborative process. 
Moreover, most of the process evaluation 
is conducted during the implementation 
and maintenance stage. 3 However, a com¬ 
munity partnership goes through other 
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important stages in its development, such 
as pre-formation and formation. 4 

The pre-formation stage includes iden¬ 
tifying potential members to recruit, con¬ 
ducting a needs assessment, and collecting 
surveillance data. The formation stage 
involves clarifying issues, recruiting mem- 
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bers, formalizing rules and procedures, 
developing policy and procedure manuals, 
clearly defining roles and expectations from 
members, developing written goals and ob¬ 
jectives, and developing an action plan and 
mission statement. 4 ' 5 Formation has been 
defined as the “initial building of the coali¬ 
tion as an organization.” 6 During formation 
key leaders and staff develop structures and 
operating procedures, build strong relation¬ 
ships and trust among the stakeholders, 
and conduct assessment and planning that 
will lead to the implementation of effective 
functioning of the coalition. 7 

Despite the importance of the formation 
stage, few studies have incorporated process 
evaluation in the initial formation of a public 
health community collaboration. 7 ' 9 Histori¬ 
cally, little emphasis has been given to evalua¬ 
tion as resources are more likely to be spent on 
interventions that are visible to stakeholders. 10 
Consequently, this study describes the first 
application of a systematic process evaluation 
of a partnership during the formation stage. 
Evaluation is usually viewed by stakeholders 
as costly and time-consuming, and it often 
ends up being a “do-it-yourself” model. 11 
This study is significant because it can help 
other practitioners and researchers involved 
in partnership and coalition development 
(especially partnerships still in the planning 
or formation stage) further appreciate the 
importance of process evaluation during all 
phases of partnership development, and will 
assist them in the planning and implementa¬ 
tion of process evaluation. 

PURPOSE 

The overall aim of this evaluation study 
was to monitor the collaborative process 
of developing an action plan for the im¬ 
provement of the health status of central 
Oklahomans through the Central Okla¬ 
homa Turning Point (COTP) initiative, by 
answering the evaluation questions posed 
by the advisory committee and assessing 
if the meeting objectives were met. Process 
evaluation was conducted by utilizing a par¬ 
ticipatory action research approach. 12 Par¬ 
ticipatory evaluation, a form of participatory 
action research, is a partnership approach 


to evaluation that engages the community 
stakeholders as co-researchers in all aspects 
of the evaluation design, implementation 
and interpretation of the results. 13 Evaluators 
and community participants work collab- 
oratively to define outcomes and measures as 
well as collect process and outcome data, and 
analyze findings. Decision making is a joint 
responsibility between the evaluator and the 
lay participants throughout the evaluation, 
although the level of participation of differ¬ 
ent stakeholders may differ at various stages 
in the evaluation. For instance, in this study 
the stakeholder-members of the evaluation 
team were primarily engaged in the design 
of the process evaluation and interpretation 
of interim results. 

METHODS 

Description of the Collaborative Process 
and a Historical Perspective 

Oklahoma has consistently ranked 
toward the bottom of national health rank- 
ings. 14 Despite efforts to reverse these trends 
during the 1980s and the 1990s, Oklahoma’s 
health status indicators failed to improve. 
Therefore, the local and state public health 
officials decided to rethink the delivery of 
public health by emphasizing the use of state 
and local collaborative partnerships in order 
to identify intervention priorities from com¬ 
munity partners. Turning Point, a program 
funded by The Robert Wood Johnson Foun¬ 
dation and the W.K. Kellogg Foundation, 
facilitated this new approach. There are cur¬ 
rently 66 partnerships in Oklahoma based 
on the Turning Point philosophy, which 
are in varying degrees of development. The 
collaborative work of these partnerships 
has resulted in the development of various 
health promotion initiatives and sustained 
community system changes. 15 

The COTP initiative was launched by 
a group of community leaders to identify 
and prioritize the health problems in cen¬ 
tral Oklahoma and ultimately create an 
action plan to address these issues. During 
the pre-formation stage of the partnership, 
208 stakeholders were recruited. During the 
planning process, the stakeholders met twice 
a month for approximately nine months 


beginning September 2003 to May 2004. A 
total of 14 meetings took place. All meetings 
took place in the evening. The collaborative 
process was divided into four main phases: 
understanding challenges and current real¬ 
ity regarding health in Oklahoma; defining 
a vision of health in Oklahoma and key 
performance areas (areas in which high 
performance is essential in order to achieve 
the vision); defining strategies; and mov¬ 
ing to action. The evaluators systematically 
shared the evaluation results at the end of 
each meeting and each collaborative phase 
with the steering committee and the stake¬ 
holders via oral reports and newsletters. 
This evaluation study was approved by the 
University of Oklahoma Health Sciences 
Center Institutional Review Board. 

Planning of the Evaluation Process 

The primary planning framework sug¬ 
gested by Steckler and Linnan 2 was used in 
the design of the process evaluation. The 
framework describes a systematic approach 
to process evaluation by providing defini¬ 
tions of key process evaluation components 
and key steps in designing and implement¬ 
ing effective process evaluation efforts. 
The key process evaluation components 
include context, reach, dose delivered, dose 
received, fidelity of the intervention, and 
implementation. 

The secondary guiding framework used 
was the Community Health Governance 
(CHG) model by Lasker and Weiss. 16 The 
CHG model hypothesizes that leadership 
and management influence the success of the 
collaborative process by determining who 
is involved in the process, how participants 
are involved and the scope of the process. 
These process characteristics, in turn, de¬ 
termine the extent to which collaboration 
can achieve the three proximal outcomes 
of collaboration, (i.e., individual empower¬ 
ment, bridging social ties and synergy), and 
thus attain the distal outcomes of the col¬ 
laboration (i.e., strengthening community 
problem-solving and improving commu¬ 
nity health). The CHG model can help the 
evaluators ask the right research questions 
pertaining to each component such as what 
the role of leadership is in the collaborative 
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process, how the stakeholders are involved in 
the process, what the scope of the partner¬ 
ship is, and how the collaborative process 
affects the stakeholders. By answering these 
questions, the evaluation team can provide 
feedback to the management and leadership 
of the coalition and make the necessary 
changes that will lead to a more effective col¬ 
laborative process. A stronger collaborative 
process will further strengthen the problem¬ 
solving ability of the community members 
and ultimately improve the community’s 
health. A visual representation of the model 
can be found in Figure 1. 

A process evaluation component matrix 
was developed that listed each component 
of process evaluation dimensions and 
related questions to be evaluated and how 
each dimension would be measured (Table 
1). The evaluation team consisted of seven 
members including the lead facilitator, rep¬ 
resentatives of the broader stakeholder 
community, and members of the steering 
committee. Additional information about 
the planning of the process evaluation can 
be found elsewhere. 17 

Data Collection and Measurement 

This evaluation employed both qualita¬ 
tive and quantitative methodology. Regard¬ 
ing quantitative research, the primary data 
collection tool was the Working Together- 


Profile of Collaboration Survey (PCS). 18 The 
PCS consists of 40 statements accompanied 
by a five-point Likert-type response scale 
ranging from “strongly disagree” (code 1) to 
“strongly agree” (code 5). This stakeholder 
satisfaction survey has established validity 
and reliability and has been administered 
to at least 23 collaborative groups prior to 
this study. Authorization to use the tool was 
given by its developers. 18 The PCS consists 
of five scales. Table 2 provides the definition 
of the constructs, the number of items, a 
representative item, and the Cronbach alpha 
for each scale. Cronbach alpha varied from 
0.65 to 0.86, which signifies an overall reli¬ 
able instrument. 

The PCS was self-administered at three 
data collection points (i.e., T1-beginning, 
T2-midpoint and T3-end of the collab¬ 
orative process) and study participants were 
unidentifiable. Participant demographic 
information was collected only during the 
last data collection point. 

Regarding qualitative research, two 
rounds of individual semi-structured inter¬ 
views were conducted with stakeholders. The 
first round took place approximately one- 
third of the way through the collaborative 
process. Fourteen interviews were conduct¬ 
ed. The method of participant selection was 
stratified random sampling to ensure that 


minority populations were well represented, 
and that a variety of opinions were expressed 
based on the degree of involvement of each 
stakeholder. In addition, stratified random 
sampling increased the credibility of the se¬ 
lection process and reduced suspicion about 
why certain stakeholders were selected for 
the interviews. 19 

The participants were divided into two 
main categories: Caucasians and others, that 
is participants representing the major ethnic 
minorities in Oklahoma (African Ameri¬ 
cans, Hispanic, American Indians and Asian/ 
Pacific Islanders). Within each category, 
the participants were further divided into 
three sub-categories based on their level of 
participation for the first part of the plan¬ 
ning process, that is, frequent attendees (i.e., 
attended the first meeting and missed two 
or fewer meetings), average attendees (i.e., 
attended the first meeting and missed three 
meetings) and rare attendees (i.e., attended 
the first meeting and missed four or more 
meetings). Each participant was assigned a 
number. Three participants were randomly 
selected from each category. Among the 18 
participants asked to be interviewed, 14 
agreed, representing stakeholders in each 
sub-category. The second round of inter¬ 
views took place at the end of the collabora¬ 
tive process. Ten interviews were conducted 


Figure 1. Model of Community Health Governance (CHG) 
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SOURCE: From Lasker RD, Weiss ES. Broadening participation in community problem solving: a multidisciplinary model to support collaborative practice 
and research. / Urban Health. 2003;80(l):14-47. Reprinted with permission from the editor of Journal of Urban Health. 
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Table 1. Process Evaluation Component Matrix 

Component & Definition 1 

What will be evaluated/Related research 
questions 

How will be measured 

Context 

(Aspects of the larger so¬ 
cial, political, and economic 
environment that may have 
influenced intervention imple¬ 
mentation.) 

A) Meeting facility accessibility 

B) Overall collaborative atmosphere 

C) Quality of meeting facilities 

D) Visibility of COTP/How much COTP 
was exposed to the media? 

E) Community readiness 

A: Stakeholder Interview (SI) 

B: Interactive Group Evaluation Form 
(IGEF), Observation Form (OF), 

Profile of Collaboration Survey (PCS), 
Steering Committee Feedback (SCF) 

C: IGEF SI, SCF 

D: Media logs 

E. PCS 

Recruitment 

(Procedures used to approach 
and attract participants.) 

A) Visibility of COTP 

B) What motivated stakeholders to par¬ 
ticipate in the first place? 

A: Event logs, Media logs 

B: SI 

Reach 

(Proportion of intended target 
audience that participated in 
an intervention.) 

A) Percent of intended stakeholders who 
participated in each meeting 

B) Percent of intended stakeholders who 
participated in each meeting based 
on race/ethnicity and affiliation 

A & B: A sign-up sheet was used for ev¬ 
ery meeting. A data-base stakeholder 
descriptive was used to identify 
group representation 

Dose delivered 

(Number or amount of intend¬ 
ed units of each intervention 
or each component delivered 
or provided.) 

A) Number of meetings that took place 

B) Length of meetings 

C) Content/topics covered in each meet¬ 
ing 

A & B: Agendas, OF 

C: Facilitator check-off list, OF 

Dose received 

(Extent to which participants 
actively engaged with, inter¬ 
acted with, were receptive 
to, and/or used materials or 
recommended resources.) 

A) New knowledge and skills gained by 
stakeholders 

B) Meeting Objectives achieved/ Did 
the participants reach a consensus (in 
each meeting) if that was the intent of 
the meeting? 

C) How well did the participants work in 
their working groups? 

A: SI 

B: Random Electronic Survey (RES), OF 

C: OF SI 

Fidelity 

(Extent to which the interven¬ 
tion was delivered as planned.) 

A) Did collaboration process take place 
as planned (meetings, content, objec¬ 
tives)? 

B) Are we staying true to the process 
and principles of workinq toqether? 

A & B: Review of original plan of meet¬ 
ings, agendas, sign-up sheets, IGEF 

OFSI 

Barriers 

(Problems encountered in 
reaching participants) 

Why do people quit coming to 
the meetings? 

SI 

Maintenance/Retention 
(Keeping participants involved 
in the programmatic and data 
collection) 

A) What do we do to keep participants 
involved? 

B) What makes people keep coming to 
the meetings? 

C) What do we do that gets people to 
each meeting? 

A & C: Staff Interview 

B: SI 

'Steckler A, Linnan L, eds. Process evaluation for public health intervention and research. San Francisco: Jossey-Boss; 2002. 
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Table 2. Description and reliability of Profile of Collaboration Survey 

Construct 
(survey items) 

Definition 

Representative item 

Cronbach 

alpha 

(Published) 2 

Cronbach 

alpha 

(Calculated) 

Context of 

Collaboration (1-3) 

The degree of readiness of 
the collaborative group in 
terms of perceived needs, 
leadership, and capacity to 
undertake this collaborative 
effort 

Now is a good time to ad¬ 
dress the issues about which 
we are collaborating 

0.46 

0.65 

Structure of 

Collaboration (4-15) 

The design of collaboration in 
terms of membership, com¬ 
munication methods, and 
overall organization 

We have adequate staff assis¬ 
tance to plan and administer 
the collaborative effort 

0.77 

0.73 

Collaboration members 
(16-23) 

The members' skills and 
attitudes toward working 
together on this process 

Stakeholders are effective 
liaisons between the organi¬ 
zations and the group 

0.87 

0.83 

Collaboration process 
(24-34) 

The process that is being used 
to reach decisions 

Stakeholders have an effective 
decision making process 

0.85 

0.86 

Results of the 
Collaboration (35-40) 

The results that are 
being accomplished 
due to the collaborative 
process 

Stakeholders have concrete, 
measurable goals to judge the 
success of the collaboration 

0.80 

0.75 

2 Chrislip, D. D., & Larson, C. E. (1994). Collaborative leadership: How citizens and civic leaders can make a difference. San Francisco, CA: Jossey-Bass. 


using the same method of selection as the 
first round but with a modified definition 
of meeting attendance. There was repre¬ 
sentation in all but one category, the one of 
“others-average.” In addition, seven steering 
committee members, one staff person and 
two facilitators were interviewed. 

All interviews were done face-to-face 
except for one, which was done over the 
phone. Each interview lasted approximately 
30 minutes. The interviews were conducted 
by three interviewers who were previously 
trained in qualitative research. The three in¬ 
terviewers also worked together on analyzing 
the data collected. 

Other measurement tools used to moni¬ 
tor the quality of each meeting included 
attendance and media monitoring logs, 
the Interactive Group Evaluation Form, 
the Facilitator Checkoff List, the Random 
Electronic Survey and the Meeting Obser¬ 
vation Form. 20 The Meeting Observation 


Form consisted primarily of 30 items, each 
of which was measured by an individual 
five-point Likert-type scale with endpoints 
“not at all” to “greater extent” and assessed 
the quality of the meeting regarding the 
following aspects: (1) Design of the meet¬ 
ing; (2) Disciplinary content (i.e., whether 
the participants were able to understand 
the important ideas presented in the ses¬ 
sion); (3) Implementation of the meeting; 
(4) Participant behavior; and (5) Culture / 
Equity. The attendance monitoring logs were 
used to monitor who participated in the 
meetings, his/her racial/ethnic background 
as well as his/her professional affiliation. 
The media monitoring logs were used to 
monitor the exposure of COTP during this 
planning process and they were completed 
by a professional media consultant who was 
also a COTP stakeholder. The Interactive 
Group Evaluation Form was given at the end 
of each meeting to each task group with the 


aim to solicit answers to two basic evaluation 
questions: (1) What helped make this meet¬ 
ing work? and (2) What should be changed 
to make this meeting better? The Random 
Electronic Survey, which was developed by 
the evaluation advisory committee, was 
sent electronically to a random sample of 
meeting attendees the day after the meeting. 
Its purpose was to assess whether all of the 
intended meeting objectives were achieved 
through consensus by the end of each meet¬ 
ing. Finally, the Facilitator Checkoff List was 
filled out at the end of the meeting by the 
lead facilitator who indicated (from a scale 
of 0 to 3, with 3 being completely covered) 
the degree to which he had covered the topics 
or objectives of the meeting. 

Data Analysis 

The quantitative data analysis consisted 
primarily of descriptive statistics for data 
collected during all collection points. Ad¬ 
vanced stage statistics (i.e., bivariate and 
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multivariate analyses) were only conducted 
for the data collected during the last data 
collection point in order to answer the ques¬ 
tion “What makes people keep coming back 
to the meetings?” Bivariate analysis was used 
to test the association between PCS variables 
and demographic variables. Logistic regres¬ 
sion was used to assess the relationship 
between meeting attendance and possible 
covariates. For the analysis, attendance was 
dichotomized into those who attended six 
or fewer meetings as opposed to those who 
attended more than six meetings. The pos¬ 
sible covariates included demographic and 
summary collaboration variables. Sum¬ 
mary collaboration variables were created 
by calculating the mean response of each 
individual for the five collaboration catego¬ 
ries. The logistic regression was conducted 
in a manual stepwise fashion. The variables 
were added to the model one by one to 
determine if they were associated with at¬ 
tendance using an alpha of 0.05. If they did 
not meet the criterion they were dropped 
from the model and the next variable was 
assessed. Variables not in the final model 
were also assessed as possible confounders. 
Microsoft Excel was used for data entry and 
conducting the descriptive statistics during 
the first and second data collection points. 
SAS version 9.1 was used for the advanced 
statistical analysis. 21 

Regarding qualitative research, the inter¬ 
views were transcribed by a staff member. 
One of the evaluators re-listened to each 
tape while reading the respective transcript 
and verified that the transcript provided 
truly represented the discussion; if necessary, 
each transcription was edited. Two evalua¬ 
tors and one staff member who had been 
trained in qualitative data analysis coded 
the transcriptions independently. Once the 
coding was completed, each coder grouped 
similar codes together in order to identify 
major themes or concepts. 22 The coders then 
compared their results in order to reach a 
consensus and identify important themes 
across all participants. Themes were said 
to be related and important if they were 
mentioned more than once by at least half 
of all the interviewees. 


RESULTS 

Attendance, Racial/Ethnic 
Representation and Stakeholder 
Affiliation Representation 

Of 208 invitees, 146 stakeholders at¬ 
tended the first meeting. Attendance de¬ 
clined over the next meetings (the lowest 
attendance was 74) and leveled off by the 
fifth meeting to reach an average of 99 par¬ 
ticipants in each meeting. Throughout the 
meetings, Caucasian attendance exceeded 
70% and African American attendance was 
approximately 15%. Hispanic representation 
was approximately equal to their representa¬ 
tion in the population residing in Oklahoma, 
particularly after the mid-point of the col¬ 
laborative process with percentages above 
5%. On the other hand, American Indians 
and Asian/Pacific Islanders were under¬ 
represented throughout the collaborative 
process. Stakeholders from the health sector 
were the predominant group represented 
throughout the collaborative process (i.e. 
their representation ranged from 25% to 
35% based on the time of the assessment), 
followed by those representing the business 
(15 % to 20%), education (10% to 14%), 
and governmental sectors (8% to 12%), 
followed by youth (3% to 5%), and finally 
those who were underserved or disadvan¬ 
taged citizens such as homeless (2%). Reach 
(i.e., the proportion of the participants who 
attended each meeting in relation to the 
number of those invited) ranged from 38% 
to 70% with an average of 47% throughout 
the collaborative process. 

Working Together-Profile 
of Collaboration Survey 

Results by PCS scale showed that among 
all scales there were no meaningful differ¬ 
ences in the mean scores during the three 
data collection points {N= 62 for Tl, N= 64 
for T2 and N = 87 for T3). For instance, in 
regards to the “Structure of Collaboration” 
construct, the mean scores (in a scale of 1-5) 
at the three data collection points were 4.03, 
4.07 and 4.06 at Tl ( beginning), T2 (mid¬ 
point), and T3 (end-point) respectively. 
Similar patterns were observed in regards 
to the other scales. Overall, the “Context of 


Collaboration” scale received the highest 
mean score for all three assessments (4.56), 
followed by “Structure of Collaboration” 
(4.05), “Collaboration Process” (4.0), “Col¬ 
laboration Members” (3. 91), and “Results 
of Collaboration” (3.81). The statements 
“The stakeholder membership includes 
those affected by the issue” and “The stake¬ 
holder membership is not dominated by 
any group or sector” received the lowest 
means scores (3.30 and 3.60 respectively). 
On the other hand, the statements that re¬ 
ceived the highest mean scores were “Now 
is a good time to address the issues about 
which we are collaborating” and “The situ¬ 
ation is critical so we must act now” (4.68 
and 4.60 respectively). 

Additional descriptive information was 
obtained during the last data collection 
point. Fifty-one percent of the stakehold¬ 
ers were female; with the majority (70 %) 
being 45-84 years old followed by 28% at 
25-44 years old and only 2% under 25 years 
old. The majority of the stakeholders were 
Caucasian (62%) followed by 12% African 
American, 4% American Indian, 2% His¬ 
panic, and 2% Asian/Pacific Islanders. The 
majority of the participants participated in 
more than 10 meetings (61%), 24% partici¬ 
pated in 7-9 meetings and 15% participated 
in less than six meetings. 

The results of the bivariate analysis 
did not show any statistically significant 
association between the PCS variables and 
any of the demographic variables. More¬ 
over, logistic regression indicated that the 
construct “Collaboration Membership” 
was the only PCS variable associated with 
attendance with an odds ratio of 3.35 
(95% Cl = 1.08,10.37, P = 0.035). In other 
words, the odds of having high attendance 
(7-13 meetings) are 3.35 times greater for 
every one point increase in the rating of 
collaboration membership. 

Semi-structured Interviews 

Twenty-four stakeholders participated to 
the one-on-one interviews. Of those, 54% 
were Caucasians and the rest represented 
the major ethnic minorities of Oklahoma 
(i.e., African American, American Indian 
and Hispanic). Moreover, 41% of the inter- 
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viewees were characterized as frequent, 25% 
as average and 34% as rare attendees. 

Stakeholders overall felt very positive 
about their experience with the collabora¬ 
tive process. Some reasons for this positive 
experience include the opportunity to 
listen to different individuals’ opinions, the 
stakeholders’ involvement in a diverse group 
of individuals, the anticipation that this 
process would lead to a positive result, the 
opportunity to contribute to a discussion, 
the well-structured collaborative process, 
and the respect the individuals extended to 
each other. When asked about the strengths 
of the collaborative process, diversity of 
stakeholders was an overarching theme. 
Diversity referred to culture, background 
and career fields. 

Interviewees also noted as strengths the 
well-planned meetings, the structured pro¬ 
cess of the collaborative initiative, the skilled 
facilitation of the meetings, the open com¬ 
munication lines and meaningful dialogue 
among the stakeholders, the dedication of 
the staff to this process, and the effective 
leadership of COTP. On the other hand, the 
interviewees perceived as weaknesses the 
length of the meetings (three hours long), 
the length of the collaborative process (nine 
months), the under-representation of all 
community groups, including the ethnic/ 
minority populations and low-income 
populations, and the lack of professional 
facilitators within each task group. 

When stakeholders were asked how 
the collaborative process affected them, 
some stakeholders stated that through this 
collaborative process they gained greater 
knowledge regarding the health issues in 
Oklahoma and expanded their networking. 
Others said that the process helped them 
improve their listening skills and become 
more sensitive and tolerant to others’ points 
of view. For others, the collaborative process 
gave them a chance to express themselves 
and share their opinions. Moreover, during 
the second round of interviewing, a few 
stakeholders also stated that by participat¬ 
ing in this process they became more health 
conscious in regards to their eating and 
exercising habits. 


When stakeholders were asked what 
motivated them to come back to the meet¬ 
ings, they acknowledged that the severity 
of the public health issues in Oklahoma as 
well as their personal commitment to this 
process were the primary reasons for their 
continued interest in the process. Others 
also felt that the process was productive and 
moved the group toward achieving project 
goals. Stakeholders also reported the reasons 
that made them initially participate in the 
collaborative process, including a personal 
invitation to participate by someone they 
knew and highly respected, the magnitude 
of the effort itself in terms of bringing such 
a large number of stakeholders together, 
and the leadership of the COTP creating an 
atmosphere where stakeholders felt equal 
to each other. 

Leadership was a subject that received 
special attention during the second round 
of interviews. Most of the interviewees 
described the characteristics of good leader¬ 
ship as someone being energetic, respected 
by the community, passionate, humble, and 
committed to the collaborative process. The 
majority of the interviewees identified the 
two co-chairs as the leaders; however, the 
steering committee, the main facilitator, and 
the group task facilitator were also identified 
as leaders. The interviewees praised the fact 
the steering committee was able to facilitate 
this collaborative process by providing the 
necessary resources and without influencing 
the stakeholder decision-making process. 

During the last phases of the collab¬ 
orative process (i.e., defining strategies, and 
moving to action), the stakeholders spent 
most of their time working in their assigned 
task groups. All of the participants enjoyed 
working in the task groups. Some of the 
challenges the stakeholders faced included 
the need for additional meetings outside 
of those regularly scheduled and the need 
to conduct research on workgroup topics, 
which felt overwhelming for some. More 
staff support was suggested to conduct 
background research for group members 
and support workgroup activities. 

As said earlier, seven steering committee 
members, one staff person and two facilita¬ 


tors were also interviewed. Some important 
recommendations given by the interviewees 
to better enhance the collaborative process 
included the re-definition of the role of the 
co-chairs of the steering committee (i.e., 
to oversee the whole process rather than 
conduct group facilitation), the need for 
continuous recruitment for new members 
and leaders, the development of a repository 
of information (gathered by the members 
of the task groups) that can be used for re¬ 
search, and the identification of a permanent 
structure in terms of staff and office location 
for COTP. 

Meeting Observations 

Both of the evaluators were present at 
the meetings; their role was to observe and 
rate the quality of the collaboration among 
stakeholders during each meeting.With the 
use of the Meeting Observation Form, the 
overall mean scores for all 14 meetings, on 
a scale of 1-5 (with 5 indicating the highest 
quality) are the following: Design of the 
meeting (4.82), Disciplinary content (4.88), 
Implementation of the meeting (4.89), Par¬ 
ticipant Behavior (4.70), and Culture and 
Equity (4.90). The evaluators’ also noted 
on the logs that the stakeholders seemed 
to enjoy engaging in networking before the 
beginning of the meeting. During the task 
group discussions the observers noted that 
the facilitator encouraged open participa¬ 
tion and paraphrased when necessary. The 
stakeholders were attentive, asked questions 
and were well-engaged in the discussions 
that were taking place. A summary of all the 
results in relation to each research question 
can be found in Table 3. 

DISCUSSION 

The purpose of this evaluation study was 
to monitor the process of developing an 
action plan toward the improvement of the 
health status of central Oklahomans through 
the COTP, by answering the evaluation ques¬ 
tions posed by the advisory committee and 
assessing whether the meeting objectives 
were met. Process evaluation reports at the 
end of each meeting and interim reports 
at the end of each collaborative phase were 


136 American Journal of Health Education — May/June 2011, Volume 42, No. 3 



Eleni L. Tolma, Marshall K. Cheney, David D. Chrislip, Derek Blankenship, Pam Troup, and Neil Hann 



provided to the steering committee and the 
lead facilitator as well as all the stakehold¬ 
ers via a newsletter. The steering committee 
and the lead facilitator used the results to 
enhance the quality of the collaborative 
process, and keep stakeholders engaged in 
the process. The fact that an action plan 
(the outcome of this planning process) is 
in place and currently in use by COTP is an 
indication of the emphasis that was given by 
the leadership in assuring a strong formation 
stage. Ultimately, a comprehensive and an 
in-depth evaluation process contributed to 
strengthening and promoting the sustain¬ 
ability of this partnership. 

Undoubtedly, stakeholder participation 
in community partnerships is important in 
building capacity and promoting sustain¬ 
ability of the partnership. 11 One of the main 
findings of this study was that reach in terms 
of attendance was relatively low (-50%) 
ranging from 37% to 70%. Similar results 
were obtained in another community- 
based intervention, during which reach 
ranged from 43% to 100%. 23 Reach was also 
weak in terms of racial/ethnic and career 
background representation. These findings 
were not surprising. Difficulties in the re¬ 
cruitment, reach, and retention of a diverse 
group of stakeholders in community-based 
interventions have been noted in the litera¬ 
ture. 7 One way to overcome this weakness 
is to incorporate stakeholders representing 
various segments of the population in the 
steering committee so that they not only 
contribute to the implementation of tasks 
already placed on the agenda, but also con¬ 
tribute to the setting of the agenda. 9 The 
importance of stakeholder diversity was 
highlighted through qualitative research 
since diversity was identified by stakehold¬ 
ers as both a strength and a weakness of the 
collaborative process. As a strength, diversity 
was defined in terms of race/ethnicity and 
career background. As a weakness, diversity 
was defined by underrepresentation of lower 
socioeconomic strata and the business sec¬ 
tor. Kreuter et al. 4 also noted that coalitions 
sometimes focus on descriptive repre¬ 
sentation in terms of social/demographic 
characteristics where the members selected 


have little accountability to their groups, in 
contrast to substantive representation. 

This study also provided some insights 
as to why stakeholders initially participated 
in the process. The majority of the intervie¬ 
wees indicated that the main reason they 
initially participated in the partnership was 
a personal invitation by someone they knew 
and highly respected, or as one stakeholder 
said “the right person asked me to.” There¬ 
fore, from a practitioner’s perspective, it is 
important during the recruitment phase that 
key people in the community (i.e., respected 
individuals with high visibility) personally 
invite the stakeholders rather than delegate 
the role to the staff. 

Participants also provided answers as to 
what can be done to sustain their participa¬ 
tion. One reason noted by the majority of 
the stakeholders regarding their continu¬ 
ous participation was the commitment the 
stakeholders felt to the collaborative process. 
As practitioners, we need to identify ways 
to keep the stakeholders committed to and 
involved in collaborative partnerships. One 
way to strengthen the stakeholders’ commit¬ 
ment is by assigning them specific tasks/roles 
during the assessment of the health issues as 
well as the identification of solutions. For 
instance, in this project, the stakeholders 
were assigned to small task groups. Recent 
research has shown that working in small 
groups and having a stable group of core 
members has been a characteristic of suc¬ 
cessful partnerships. 24 In this study, we went 
one step further to identify what would 
enhance working relationships within these 
small groups. Some of the results include 
meeting structure and good facilitation, ef¬ 
fective communication among stakeholders, 
as well as diversity in group membership. 

In addition, the finding that the col¬ 
laboration membership was the only variable 
associated with high stakeholder attendance 
corroborates other research that documents 
the importance of building strong collab¬ 
orative skills among the stakeholders. 4,24, 
25 Organizers need to provide training to 
stakeholders on collaborative skills not only 
at the onset of the partnership development, 
but also throughout the process in order to 


further strengthen stakeholder commitment. 
Other strategies to promote continuous 
participation include the strengthening of 
the coalition processes and the building up 
of the coalition structure. Literature also 
suggests that stakeholders who perceive 
more benefits than costs are more likely to 
stay committed and continue to be involved 
with the partnership. 26,27 

Another interesting finding derived from 
the quantitative research was that communi¬ 
ty readiness was the one aspect of the collab¬ 
orative process during the administration of 
the PCS that consistently received the high¬ 
est score. This is not surprising because the 
PCS was administered during the early stage 
of the partnership. When the motivation to 
develop a partnership comes in response to 
some pressing issues, in this case the poor 
health status of Oklahoma, then community 
ownership is enhanced and there is greater 
likelihood that the partnership will be sus¬ 
tained. 27 Practitioners who are considering 
launching a community partnership should 
first assess the community’s readiness before 
implementing such a project. 

Moreover, the importance of leader¬ 
ship in successful partnerships has been 
highlighted in this study as well as in the 
literature. 4,24,28 Effective leadership has been 
characterized as collaborative leadership, the 
one that effectively facilitates productive 
interactions among partners, by bridging 
diverse cultures, sharing power, facilitating 
open dialogues, and resolving conflicts. 28 

The whole planning collaborative process 
was an educational and liberating process for 
the stakeholders through building critical 
consciousness. 29,30 The stakeholders were 
able to examine in-depth the health issues 
of Oklahoma that affected their lives, reflect 
upon those issues, engage in an authentic 
dialogue, and suggest plans of action or 
solutions to the identified issues. The pro¬ 
cess evaluation revealed evidence of this 
liberation process. For instance, during the 
first round of interviews, the participants 
acknowledged that the planning process 
helped them realize the magnitude of the 
health issues in Oklahoma, and identify 
the root causes of bad health. Participants 
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Table 3. Summary of the Results of Process Evaluation 

Research Question 

Results 

1 . What motivated stakeholders to partici¬ 
pate in the first place? 

■ Personal invitation by someone they knew and highly respected 

■ The magnitude of the effort of bringing a large number of stakeholders 
together 

■ Leadership of the Central Oklahoma Turning Point (COTP) 

■ Community readiness to undertake this project 

2. What makes stakeholders keep coming 
back to the meetings? 

■ Commitment felt by the stakeholders to the collaborative process 

■ Severity of public health issues in Oklahoma 

■ Productive meetings 

■ Curiosity what the outcome would be 

■ Enhanced collaborative skills 

3. What do we do to keep stakeholders 
involved in this process? 

■ Enhanced communication system (e.g. use of internet) 

■ Strong organizational structure (e.g. skilled facilitation, participatory na¬ 
ture of meetings, productive meetings, open communication channels) 

■ Democratic collaborative process based on the principles of respect and 
fairness 

■ Incentives (e.g. health screenings and opportunities for networking and 
education) 

4. Have the meeting objectives been 
achieved? 

■ More than two-thirds of the respondents of the Electronic Random Sur¬ 
vey in each meeting agreed that the objectives for each meeting were 
achieved 

5. Has a concrete plan of action been 
achieved? 

■ A report was developed which outlined the stakeholders' findings and 
the nine strategies to improve the health in Central Oklahoma region 

6. Did the stakeholders reach a consensus 
if that was the intent of the meeting? 

■ Based on the Electronic Random Survey results consensus was consis¬ 
tently achieved 

7. Are we staying true to the process and 
principles of working together? 

■ Based on the Electronic Random Survey results, stakeholder interviews 
and observations by the evaluators it was confirmed that the stakeholders 
followed the principles of working together 

8. Have all the meetings taken place as 
planned and has the content of each 
meeting been covered? 

■ All meetings took place as planned 

■ All topics of each meeting were covered based on the Facilitator Checkoff 

List 

9. What are the strengths and weaknesses 
of the collaborative process? 

■ Strengths: 

■ Diversity (race/ethnicity and career background) 

■ Well-planned meetings 

■ The structured process of the collaborative initiative 

■ The skilled facilitation of the meetings by the lead facilitator 

■ The open communication lines and meaningful dialogue among the 
stakeholders 

■ The dedication of the staff to this process 

■ The effective leadership of COTP 

■ Weaknesses: 

■ Diversity (not having enough representatives from lower socioeconom¬ 
ic strata or representatives from the business sector) 

■ The length of the meetings (three hours long) 

■ The length of the collaborative process (nine months) 

■ The lack of professional facilitators within each task group 

Continues on next page 
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Table 3. Summary of the Results of Process Evaluation (Con't) 

Research Question 

Results 

What role did leadership play in this col¬ 
laborative process? 

■ Strength of this collaborative process 

■ Provided necessary resources without influencing decision-making process 

What contributed to the cohesiveness or 
lack of cohesiveness of each working com¬ 
mittee? 

■ Cohesiveness: 

■ The need to achieve a common goal 

■ The topic assigned was specific 

■ The discussions took place within an environment of respect 

■ The process was structured and had good facilitation 

■ The use of e-mail as an effective method of communication among 
the stakeholders 

■ Lack of cohesiveness 

■ The topic was too broad to be examined in-depth 

■ The core task group consisted of a small number of individuals with 
limited expertise and skills 

■ The lack of a professionally trained facilitator 

Why people quit coming to the meetings? 

■ Personal conflicts 

■ Commitments to other events 

How much was COTP exposed in the 
media? 

■ Continuous coverage throughout the collaborative process 

■ Highest coverage in September (launching of COTP), December (Polling, 
Town Hall meetings and "listening sessions"), and June (announcement 
of recommendations and action plan) 


also mentioned how they have engaged in 
a meaningful dialogue, during which they 
felt their voices were heard, and that they 
felt equal to each other. During the second 
round of interviews we observed not only 
a change in their awareness regarding the 
health issues in Oklahoma, but also a per¬ 
sonal behavioral change toward healthier 
habits. These results indicate an accom¬ 
plishment of empowerment at least on an 
individual basis. 

From a methodological perspective, a 
major strength of this study was the achieve¬ 
ment of triangulation, which increased the 
accuracy and credibility of our findings. 
Triangulation was achieved by using a va¬ 
riety of data sources (surveys, observations, 
interviews), by incorporating a broad-based 
evaluation team, and by combining mixed 
methodology. 31 For example, the under¬ 
representation of stakeholders from ethnic 
minority groups and other disadvantaged 
groups was identified as a theme throughout 
the evaluation through various sources, such 
as individual interviews with stakeholders, 


attendance monitoring logs and the results 
obtained through the PCS administration. 
Triangulation can be expensive and time 
consuming; however, it is imperative that re¬ 
searchers try to achieve triangulation despite 
budget, time or political constraints. 19 

Another methodological strength was 
the use of participatory evaluation. Various 
benefits are associated with participatory 
evaluation. The most prominent ones in¬ 
clude building the community’s capacity 
to conduct their own systematic data col¬ 
lection and enhance their understanding of 
evaluation. These strengths can lead to the 
application of evaluation methods in their 
own projects with appropriate training, 
facilitation and technical assistance. In addi¬ 
tion, the experiential wisdom of community 
leadership can ensure that evaluation ques¬ 
tions are important, data collection methods 
are realistic and findings are relevant and 
applicable within the local cultural context. 13 
In this study, the stakeholder-members of 
the evaluation team provided invaluable in¬ 
sights as to what to evaluate, how to evaluate 


it, how to interpret the results and how to 
disseminate them. For example, the question 
in the interview guide concerning the cohe¬ 
siveness or lack of cohesiveness of the task 
group in which the stakeholders belonged 
to was proposed by the community stake¬ 
holders. In addition, it was the stakeholders 
who suggested and designed the use of the 
Random Electronic Survey as a way to assess 
whether the objectives set for each meeting 
were actually accomplished. 

Moreover, the use of a systematic com¬ 
prehensive approach to evaluation based 
on the evaluation framework developed by 
Steckler and Linnan 2 as well as the theo¬ 
retical framework of the Community Flealth 
Governance model by Lasker and Weiss 16 
were instrumental in the success of this 
model. The first model helped us to opera¬ 
tionalize the concept of process evaluation 
(e.g., reach, dose delivered/received, fidelity), 
and the second one to identify the specific 
aspects related to collaborative work such 
as leadership and management. The results 
of this study showed that strong leadership 
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and management are important factors to 
reach (at least) the proximal outcomes of 
collaboration such as individual empower¬ 
ment or bridging social ties. 

On the other hand, there were also some 
methodological weaknesses. One challenge 
encountered during the process evaluation 
related to the administration of the PCS. 
During the three data collection points, not 
all participants were the same at each time of 
the data collection and moreover, the partici¬ 
pants consisted of a convenience sample and 
not of a random sample as may be expected 
in a trend longitudinal study.“Although this 
might seem as a limitation of the study, one 
has to keep in mind that it was impossible 
to have control over who participated in 
the research study since the evaluators had 
no control over who attended the meetings. 
Stakeholders were volunteers who donated 
their personal time after regular work hours 
to attend these meetings. This meant that 
evaluators needed to be respectful of the 
stakeholders’ time and of the fact that they 
were probably mentally and physically tired 
by the time they conducted the evaluation 
survey (i.e., the PCS) at the end of the meet¬ 
ing. To enhance completion of the survey, 
the time needed to complete the PCS was 
kept to less than 10 minutes. 

The lack of meaningful differences in the 
mean scores for each PCS scale during its 
administration at three different points of 
time might suggest that the tool was not 
sensitive enough to measure how the stake¬ 
holders truly felt about the collaborative 
process. Despite this limitation, the results 
of the PCS have served as a “snapshot” at 
three different points of time of how the 
stakeholders felt about the collaborative 
process. Evaluators working in the area of 
community development need to be aware 
of these methodological limitations and if 
possible be knowledgeable of the principles 
of community organization and develop¬ 
ment prior to engaging in such a project. 
Rigorous evaluation methods may be dif¬ 
ficult to implement because community 
work is a heuristic process. Community- 
based research is action-research and in¬ 
novative approaches often need to be taken. 


Stakeholders might not be interested in 
research and traditional instruments might 
not be conducive to achieving the goals of 
the collaboration. 

TRANSLATION TO HEALTH 
PROMOTION PRACTICE 

There are numerous recommendations 
that can be derived from this study; how¬ 
ever, we will focus on the following that we 
consider as the main ones: 

• Diversity (in terms of ethnicity, back¬ 
ground and demographics) was identified 
as a major strength as well as a weakness. 
Therefore, more effort should be given in 
encouraging ethnic minority populations 
to engage in the collaborative process. Other 
community groups that can be characterized 
as powerless or marginalized but primarily 
affected by the health problems, such as 
youth or homeless people, should also be en¬ 
couraged to participate. Moreover, diversity 
within the coalition membership needs to be 
continually monitored and if necessary, the 
leaders of the coalition enhance their efforts 
of recruitment. 

• Coalitions need to continue promoting 
stakeholder participation and commitment 
by engaging all stakeholders actively not only 
in the implementation of the action plan, 
but also in the development of the action 
plan, through an open dialogue and based 
on the democratic principles of respect and 
equality. Active stakeholder engagement 
and participation can enhance community 
capacity, which can further lead to sustain¬ 
able community changes. 33 

• Promoting continuous stakeholder par¬ 
ticipation is an important aspect of building 
capacity of a newly formed coalition. The 
results of this study indicate that building 
collaborative skills at the early stage of the 
partnership formation and throughout the 
collaborative process could be an effective 
way of enhancing participation. Another 
way is by reinforcing the stakeholders’ com¬ 
mitment to this process. This can be done 
by providing incentives for community 
participation such as health screening and 
opportunities for networking. Stakeholders 
who perceive more benefits than costs are 


more likely to stay committed and continue 
to be involved with the coalition. 

• Process evaluation is an important tool 
for continuous monitoring of the qual¬ 
ity of the collaborative process during the 
planning phase of a partnership. Minimum 
process evaluation measures should be incor¬ 
porated in the evaluation plan. 

• Community readiness was an essential 
prerequisite of this successful collaborative 
initiative and therefore, organizers of similar 
initiatives need to take this element in con¬ 
sideration during the planning process. 

• Effective leadership is an important as¬ 
pect of success of any collaborative process 
and efforts should take place to maintain a 
high quality of leadership. 

• Coalitions need to continue to col¬ 
laborate with media representatives in order 
to enhance their visibility and community 
participation. 

• Coalitions need to continue to foster 
communication among their members 
through newsletters, emails, websites, and 
other similar tools. 

In conclusion, no published study has 
described the application of a systematic 
process evaluation of a partnership dur¬ 
ing the formation stage. The results of this 
process evaluation provide support to the 
existing literature that indeed coalition 
processes, coalition structures, leadership 
and staff, membership, and the context and 
history of the partnership are all important 
elements for a strong formation of a part¬ 
nership. 7 In addition, this study highlighted 
the importance of building effective col¬ 
laborative skills among the stakeholders 
early in the planning phase as it relates 
strongly to high stakeholder attendance at 
the partnership meetings. Moreover, the use 
of qualitative research was instrumental in 
understanding various aspects of the col¬ 
laborative process (e.g., how the stakeholders 
felt about the collaborative process) that 
could not have been assessed via quantitative 
research. Not all collaborative partnerships 
and coalitions have the resources to con¬ 
duct such an in-depth and comprehensive 
process evaluation. However, some form of 
process evaluation during the formation 
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stage as well as during the other stages of 
development is recommended. As Butter- 
foss et al. 7 have emphasized, formation does 
matter if we want to build strong and sus¬ 
tainable partnerships. 
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